In environments where a video stream and an animation sequence are combined to be presented at the same time, accurately synchronizing the frames in the video stream with the frames in the animation sequence so that each individual video frame may be displayed together with a corresponding animation frame is required. In addition, video frames are usually composited with the corresponding animation frame to obtain a composite frame, which is then displayed.
Rendering an animation sequence using a computer often requires tremendous processing power. If the animation rendering task is given a high priority, it may take too much of the computer's processing power away from the other tasks that are being performed at the same time. For example, if a user inputs a command, the task that is responsible for handling user input may not be able to process the command and produce an appropriate response in a timely manner, because the computer processor is busy rendering the animation sequence. Accordingly, delays may result. On the other hand, if the animation rendering task is given a low priority, other processor-intensive tasks may use up too much of the computer's processing power such that the animation sequence cannot be rendered fast enough. The result is an impact on the smoothness of the animation. Ideally, the animation rendering task should be given a priority that is lower than the priority given to a very important or critical task, such as the handling of user input, but at the same time higher than the priority given to a standard yet processor-intensive task. The animation priority may also depend on how important the animation rendering is to the user.
When an animation sequence is rendered by a software program, the speed that the software program may render the individual frames forming the animation sequence depends on many variables, including but not limited to the speed and the number of the processors in the computer system, the size of the available memory, and the programming language used to program the software program. When the animation sequence is rendered in real time and is combined with a video stream so that each individual frame in the video stream is presented with a corresponding frame in the animation sequence, it is necessary to correlate the video frame and its corresponding animation frame accurately.
Often for performance reasons, the video stream is processed in parallel with the rendering of the animation sequence. For example, the software program may decode the video stream in one thread and render the animation sequence in another thread. Alternatively, a hardware video decoder may decode the video stream while a software program renders the animation sequence. It is often desirable that the software program is able to control the priority of the two threads. Most often, the video frames are decoded at a speed different from the speed at which the animation frames are rendered. Sometimes, delays in various parts of the computer system exist. For example, application programs may be unresponsive or experience pauses due to garbage collection (GC). One possible method for synchronizing the video frames and the animation frames is to pre-render the entire animation sequence ahead of time and store the individual animation frames in memory to be used at a later time. However, this does not address the situation where the animation sequence must be rendered in real time.